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Abstract Body 


Background / Context: 

Like studies that focus on detecting a main effect, a critical consideration in designing 
studies to detect moderation effects is the statistical power with which the moderation effect can 
be detected if they exist. The potential moderators of the intervention in cluster randomized trials 
(CRTs) include pretest, ethnicity, school climate, or the fidelity of implementation, which could 
be at different levels and have different distributions (e.g., binary, continuous). For moderator 
relationships in experimental studies. Bloom (2005), Spybrook (2014), and Spybrook, Kelcey, 
and Dong (2015) have presented procedures for conducting power analysis for binary moderators 
in two- to four-level cluster randomized trials (CRTs), but have not extended those procedures to 
include continuous moderator variables. Furthermore, no computational tools to facilitate use of 
these techniques by researchers have been developed. 

Most recently, Mathieu, Aguinis, Culpepper, & Chen (2012) conducted a comprehensive 
Monte Carlo simulation to estimate the statistical power to detect cross-level interaction effects, 
and Dong (2014) presented the formulas to calculate minimum detectable effect size (MDBS) for 
continuous moderator analysis in two-level CRTs with a level-2 continuous moderator. 

However, Mathieu et al (2012) only studied two-level analyses without including covariates, and 
did not provide closed form formulas to estimate the statistical power, MDBS, or minimum 
required sample size to detect meaningful effects; Dong (2014) only focused on a level-2 
continuous moderator in two-level CRTs, and didn’t study cross-level moderation, include 
covariates, or examine three-level CRTs. In sum, there is no a systematic study investigating the 
power analysis to detect moderation effects in two- and three-level CRTs that includes both 
binary and continuous moderators, same and cross-level moderation, with covariates, and 
computational tools. 

Purpose / Objective / Research Question / Focus of Study: 

The purpose of this study is to propose a general framework for power analyses to detect 
the moderator effects in two- and three-level CRTs. Specifically, we aim to: (1) develop the 
statistical formulations for calculating statistical power, minimum detectable effect size (MDBS) 
and its confidence interval to detect the moderation effects in two- and three-level CRTs, which 
include same and cross-level moderation, binary and continuous moderators, and covariates, and 
(2) operatize these formulas in the enhanced version of PowerUp! (Dong & Maynard, 2013) to 
create spreadsheets for calculating power, MDBS, etc. 

Significance / Novelty of study: 

Bducational researchers have interests in the effects of both binary and continuous 
moderators in CRTs. Statistical power analysis is appropriate in the planning stages to help 
researchers design studies with sufficient power to detect such relationships when they are large 
enough to have practical or theoretical significance. This study will provide a general framework 
and computational tool for power analyses to detect the binary and continuous moderator effect 
in two- and three-level CRTs. 

Study Design: 

This study covers the same and cross-level moderation in two- and three-level CRTs. The 
detailed study designs and analysis models are presented at Table 1 (see appendix). Bor the 
cross-level moderation, i.e., the moderator at the level lower than the treatment level, there are 
two options: the fixed slope and random slope of the moderator variable. The fixed slope of the 
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moderator variable assumes that the effeet of the moderator varies by the treatment status, but 
does not vary aeross the higher-level elusters, while the random slope of the moderator effeets 
assume that the effeet of the moderator varies by the treatment status, and varies randomly aeross 
the higher-level elusters. The power and MDBS formulas are derived for all the designs and 
models eovered in Table 1 and are applied in PowerUpL 


Statistical, Measurement, or Econometric Model: 

Due to the page limitation, we present the results of two-level CRTs with a treatment 
variable at Level 2 and a moderator at Level 1 below. 

The HLM, ineluding one treatment variable, Tj , and one level- 1 moderator, Xg , with a random 
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The interest for a moderator analysis is whether the parameter, , whieh indieates the 
conditional relationship between the average treatment effect and the moderator, is statistically 
significant. 

According to Bxpression [3.89] in Raudenbush & Bryk (2002, p.59). 


SE{f,,) = 




P(1 - P)J 


P{\ - P)J 


(3) 


where is the proportion of variance at level 1 that is explained by the level- 1 moderator 
{Xg ): RI = \- the proportion of variance between level-2 clusters on the effect 

of Xg explained by level-2 predictor (T^.): RIj = 1 - rfj|^ / Aj . o = rfj / indicates the effect 
heterogeneity for the level- 1 covariate {Xg) across level-2 units (clusters) in the model that is not 
conditional on treatment variable, Tg , which is the proportion of the variance between clusters on 
the effect of X,, to the between-cluster residual variance, crl is the variance of X., , and P is the 

proportion of clusters in the treatment group. 

The noncentrality parameter (unstandardized) is: 


'^x ~ . 
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By standardization, let + ct^ =1 and cr^ = 1, the standardized coefficient ^ = yjj , or let 
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the noncentrality parameter (standardized) is: 


\x ~ ■ 


5^P{\-P)J 


\l-Rlj)p(D + {l-Rl){l-p)ln 
The degrees of freedom is v - J -2 . p is the unconditional intraclass correlation, 
P ~ ^00 /(^oo + cr ) • 

The statistical power is: \- fi = \- P[T {J <1^} + P[T {J - 2, ] . 


(5) 


SREE Spring 2016 Conference Abstract Template 


2 



The minimum detectable effect size (MDBS) regarding the standardized coefficient is: 
MDES{\S\) = 


{\-Rl,)pco + {\-Rl){\-p)ln 


P{\ - P)J 

where, + h/? one-tailed tests with v degrees of freedom {v - J - T), and 

Mv = t^i 2 + for two-tailed tests. The 100*(l-a)% confidence interval for MDES(\5\^ is 


( 6 ) 


given by: (M, 
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P{\-P)J 
Extension to the Fixed Slope Model 

This fixed slope HLM assumes that the effect of X.. varies by the treatment status {T.), but does 
not vary across level-2 units as in the random slope model in Expression 2. 


The Level 2 model is: 


A/ = roo + YoiTj + d 


oj 
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The noncentrality parameter (unstandardized) is: = 
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By standardization, let + cr"' =1 and cr^ = 1, or let 5 = 2^1 ■ noncentrality 
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parameter (standardized) is: 2,^ = 


I S^P(l-P)Jn 
(l-i?f)(l-p) 


( 10 ) 


The statistical power is: \ - P = \- P[T {J{n tQ] + P[T (J(n - 1) - 2,2^^^) < . 

The minimum detectable effect size (MDBS) regarding the standardized coefficient is: 


MDES(S) = M^ 


(l-R^)(l-p) 


P(1 - P)Jn 

where, =t„ + t^_p for one-tailed tests with v degrees of freedom ( v = J(n - 1) - 2 ), and 
+ d-p two-tailed tests. The 100*(l-a)% confidence interval for MDES(\5\^ is 
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given by: 
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Extension to Binary Moderator 

When the level- 1 moderator, Xy , is a binary variable with a proportion of Q in one subgroup and 


(1-0 in another subgroup, Xy ~ Bernoulli(Q) : VAR(Xy) = cr^ = Qi^-Q) 

Bor the random slope model, we insert Expression 13 into Expression 4. 
The noncentrality parameter (unstandardized) is: 
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By standardization, let tIq + ct^ =1, or let 5 = 
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, the noncentrality parameter 
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d^p{\-p)J 


(standardized) is: ^ 

V (1 - Rl,)po) + (1 - )(1 - p)l{nQ{\ - Q)) 

The statistieal power is: I- P = 1- P[T {J P[T {J - 2, )<-?(,]. 

The minimum deteetable effeet size (MDBS) regarding Cohen’s d is: 


( 15 ) 
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where, = C + h/? one-tailed tests with v degrees of freedom {v - J - T), and 
Mv = t^i 2 + for two-tailed tests. The 100*(l-a)% eonfidence interval for MDES(\5\^ is 


(16) 
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For the fixed slope model, we insert Expression 13 into Expression 9. 


The noneentrality parameter (unstandardized) is: /l^x = 

2 2 Yw 

By standardization, let r + cr =1, or let = 
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(standardized) is: = 
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The statistieal power is: I- = I- P[T {J{n - -2,A^x) < to] + Pi'T (J{n - 1) - 2,/i^;^) < . 

This result is eonsistent with the results about the power formula for the binary level- 1 moderator 
in Spybrook, Keleey, and Dong (2015). 

The minimum detectable effect size (MDBS) regarding Cohen’s d is: 


MDES(S) = M^ 


(\-Rfi(\-p) 


Pil-P)Qil-Q)Jn 

where, = C + 6 /? one-tailed tests with v degrees of freedom ( v = J(n - 1) - 2 ), and 
M„ = + 6-/? two-tailed tests. The 100*(l-a)% confidence interval for MDES{\S\) is 


( 20 ) 


given by: 


(l-Rfiil-p) 


( 21 ) 
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Results and Conclusions: 

The standard error formulas in Expression 3 indicates that the standard error of the 
moderation effect estimate is not associated with the residual variance ( ) for the intercepts, 

but is associated with the residual variance at level 1 ( (7|^). This suggests that adding more 

covariates in the intercept model would not reduce the standard error or improve power to detect 
the moderation effect, which is different from the main effect analysis, however, adding more 
covariates at level- 1 that can further explain level- 1 variance would reduce the standard error and 
increase power. 

This abstract only shows the partial results in Table 1, we will present more results and 
demonstrate their application in PowerUp! in the presentation. 
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Table 1; Designs and Analysis Models of Cluster Randomized Trials Covered in This Study 


1 

2 

3 

4 

5 

6 

Levels of Clustering =Treatment 
Assignment Level 

Model Number 

Level of 
Moderator 

Slope Effeet of 
Lower-Level 
Moderator 

Distribution of Moderators 


CRT2-lf 

1 

Fixed 

Binary 

Continuous 

2 

CRT2-lr 

1 

Random 

Binary 

Continuous 


CRT2-2 

2 

NA 

Binary 

Continuous 


CRTS- If 

1 

Fixed 

Binary 

Continuous 


CRT3-lr 

1 

Random 

Binary 

Continuous 

3 

CRT3-2f 

2 

Fixed 

Binary 

Continuous 


CRT3-2C 

2 

Random 

Binary 

Continuous 


CRT3-3 

3 

NA 

Binary 

Continuous 
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